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The Education Endowment Foundation (EEF) 


The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to 
breaking the link between family income and educational achievement, ensuring that children from all 
backgrounds can fulfil their potential and make the most of their talents. 


We aim to raise the attainment of children facing disadvantage by: 
e Identifying promising educational innovations that address the needs of disadvantaged 


children in primary and secondary schools in England; 


e Evaluating these innovations to extend and secure the evidence on what works and can be 
made to work at scale; 


e Encouraging schools, government, charities, and others to apply evidence and adopt 
innovations found to be effective. 


Founded by the education charity the Sutton Trust, as lead charity in partnership with Impetus Trust, 
the EEF is funded by an initial £125m grant from the Department for Education. With investment and 
fundraising income, the EEF intends to award as much as £200m by 2026. 
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Literacy Catch-up Round 


This study was funded by the Education Endowment Foundation as one of 23 projects funded through 
a themed funding round on literacy catch-up at the primary-secondary transition. Projects funded 
within this round aimed to identify effective ways to support pupils who do not achieve Level 4 in 
English by the end of Key Stage 2. The project was one of three funded with a particular focus on 
writing. 
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Executive summary 


The project 


The project aimed to use memorable experiences and an approach called ‘Self-Regulated Strategy 
Development’ (SRSD) to help struggling writers in Years 6 and 7. SRSD provides a clear structure to 
help pupils plan, monitor and evaluate their writing. It aims to encourage pupils to take ownership of 
their work and can be used to teach most genres of writing, including narrative writing. Memorable 
experiences, such as trips to local landmarks or visits from World War Il veterans, were used as a 
focus for writing lessons. 


In this evaluation 23 primary schools and their Year 6 teachers in the Calderdale area of West 
Yorkshire were randomly allocated to receive training, from an external consultant, in the SRSD 
approach. Twelve schools were allocated to the comparison group and 11 schools to the intervention 
group. Children in the intervention schools were taught following the SRSD approach in the last six 
weeks of the summer term in Year 6 and in the first term of Year 7 at secondary school. The project 
was organised by the Calderdale Excellence Partnership. 


The study was funded by the Education Endowment Foundation as one of 23 projects focused on 
literacy catch-up at the primary-secondary transition. It was one of three programmes with a particular 
focus on writing. 


What impact did it have? 


Overall, the project appeared to have a large positive impact on writing outcomes. The overall effect 
size for writing, comparing the progress of pupils in the project to similar pupils who did not participate 
was +0.74. This effect size was statistically significant, meaning that it is unlikely to have occurred by 
chance, and can be envisaged as saying that participating pupils made approximately nine months’ 
additional progress compared to similar pupils who did not participate in the intervention. 


The approach was also effective for pupils eligible for free school meals (FSM). Whilst there appears 
to be a larger effect for FSM pupils, the difference in the interaction test is not statistically significant, 
so chance is a possible explanation for the difference observed. 


The approach had no statistically significant effect on the secondary outcomes relating to reading, 
spelling or grammar (i.e. the small differences between the groups could have occurred by chance). 
These outcomes were not the main focus of the study, but were measured as part of the same test. 


Teachers were trained in the SRSD approach by the North American developers, but, with support 
from the Calderdale Excellence Partnership team, also adapted it in some ways for an English 
context. For example, a component of the approach which focused on developing positive ‘self-talk’ 
was Anglicised. 


Intervention vs. | .2° Primary schools 
eonivel *  3secondary schools  +0.74 +9 +0.26 to +1.22 aa 
261 pupils 
Bi ano 86 pupils +1.60 +18 +0.21 to +2.98 
Meal pupils 


*Evidence ratings are not provided for sub-group analyses, which will always be less secure than overall findings. 
For more information about evidence ratings, see Appendix R. 


How secure is this finding? 


The overall finding of the study is assessed as moderately secure. This assessment is based on the 
study’s design, its size and other factors such as the level of drop-out of participating pupils. 
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The evaluation was set up as an efficacy trial to test the impact of using SRSD in conjunction with 
memorable experiences at the transition from primary to secondary school. Efficacy trials seek to test 
evaluations in the best possible conditions, but they do not seek to demonstrate that the findings hold 
at scale in all types of schools. 


Calderdale Excellence Partnership (CEP) recruited 23 primary schools and three secondary schools 
that were served by the recruited primary schools and a SRSD developer came from North America to 
deliver training. The 23 primary schools were randomly assigned by a statistician at the York Trials 
Unit so that Year 6 teachers in the 11 intervention schools’ received training to deliver the intervention 
to their Year 6 pupils. Three secondary schools agreed to honour the randomisation by allocating 
intervention and comparison school pupils into separate Year 7 classes, and SRSD continued to be 
delivered, by secondary school English teachers, to the intervention children in the first term of their 
Year 7. All children (from both the intervention and comparison groups) were tested under exam 
conditions using the Progress in English 11 (Long Form) Test developed by GL Assessment as a 
measure of general writing ability. Primary schools allocated to the comparison group were offered the 
training in the SRSD approach at the end of the trial (known as a ‘wait-list’). 


The study was well conducted with independent randomisation and a moderate sample size. Intention- 
to-treat analysis was used (i.e. pupils were compared in the groups to which they were originally 
randomly assigned), blind marking of the test papers was undertaken, and the analysis was adjusted 
for school randomisation. Attrition was 8.5% for comparison pupils and 8.0% for intervention of those 
who were eligible because they attended both the primary and participating secondary school, which 
should have resulted in relatively little bias. 


A systematic review of all previous randomised controlled trials of SRSD, largely from North America, 
has shown that this approach to teaching writing is, on average, very effective, with large effect sizes 
reported. This trial suggests that the approach can also be effective in English schools. 


To increase the security of the finding and to assess its applicability in other schools, a larger 
evaluation could be commissioned in the future. In addition, further work could done to try and 
disentangle the role and importance of the individual components of the intervention (e.g. the training 
or memorable experiences), as this was not possible within this evaluation. 


How much does it cost? 


The cost of the approach is estimated at £52 per pupil. This estimate includes training and materials 
(£60 per teacher or £2 per pupil), and the cost of memorable experiences (£50 per pupil). Estimates 
are based on aclass of 30 pupils, and on training being delivered to a group of 30 teachers. 


1. The approach had a strong positive effect on the writing outcomes of low attaining pupils at the transition 
from primary to secondary school among a sample of pupils in State schools in the West Yorkshire area. 


2. The approach had beneficial effects for both FSM and non-FSM pupils. 


3. These findings, in combination with existing evidence from the United States and elsewhere, suggest that 
the Self-Regulated Strategy Development approach has substantial promise as a literacy catch-up 


4. A larger effectiveness trial could be commissioned to test the approach on a larger scale and with other 
age groups. 


5. Teachers were trained in the SRSD approach by the North American developers, but adapted it in some 
ways for an English context. 


1 Correction issued: 17" June 2014. A previous version of this report stated that there were 12 intervention schools. 
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Introduction 


In 2013 approximately 85,000 pupils left primary school without having attained Level 4 or above in 
writing (Department for Education, 2013a). This study assessed the impact of a project which aimed to 
improve writing outcomes for pupils who had not achieved a ‘secure Level 4’ (defined as having 
achieved a Level 4b or above) prior to moving from the final year of primary school (Year 6) to the first 
year of secondary school (Year 7). The project involved memorable experiences and the use of Self- 
Regulated Strategy Development (SRSD) over the transition period. 


A substantial minority of pupils who do not reach Level 4 in English at the end of Key Stage 2 
(incorporating reading and writing) do not make the expected level of progress in secondary school 
(Department for Education, 2014). In addition, a literature review of the research into the transfer of 
pupils from primary to secondary school has identified the potential for personal, social and academic 
outcomes to ‘dip’ after the end of Key Stage 2 (KS2) leading to a large minority of pupils failing to 
make expected progress in the first year of secondary school (Galton, Gray and Ruddock, 1999). 


The study was funded by the Education Endowment Foundation, as one of 23 projects focused on 
literacy catch-up at the primary-secondary transition. It was one of four programmes with a particular 
focus on writing. 


Background and existing literature on Self-Regulated Strategy Development 


SRSD is a writing process model in which students are encouraged to plan, draft, edit and revise their 
writing. SRSD is a strategy (originating in North America), which provides a clear structure to assist 
writers and can be used for most genres of writing, including narrative writing. There are six basic 
stages of instruction and four strategies for self-regulation, which include self-monitoring and goal 
setting, thus providing pupils with ownership for improving their own writing. SRSD uses ‘heuristics’ 
which provide scaffolding of structures and devices that aid the composition of argumentative writing — 
in particular planning — which can include examining a question, brainstorming, organising and 
sequencing ideas and evaluating. 


The approach was created in the United States in the 1990s. It has been designed, and is suitable, for 
children who are aged between 8 and 14 (Andrews et al., 2006) so it is appropriate for children 
passing through the transition from primary to secondary school. A systematic review by Andrews et 
al. (2006) found that a combination of contextual factors and specific interventions based on the SRSD 
programme were necessary for successful practice in teaching and learning writing for 7-14 year olds. 
The effect sizes of individual studies, largely undertaken in North America, were very large with 
estimates in some instances exceeding 1 standard deviation between the intervention and control 
groups. For example, in a study by De La Paz and Graham (2002) of secondary school children of 
mixed attainment (aged between 12-14) the overall quality of their writing was 1.7 standard deviations 
better than the quality of writing of children in the control group 6 weeks after the intervention. 
Nevertheless, despite these promising results in a North American context it was important to 
establish whether or not such an intervention would be effective in a British setting. For this reason the 
intervention was appropriate for an efficacy trial in English schools. 


In 2012 the Education Endowment Foundation funded the University of York and Durham University to 
independently evaluate the Improving Writing Quality intervention being delivered and supported by 
the Calderdale Excellence Partnership (CEP) in 2013 and 2014. The Improving Writing Quality 
intervention provided memorable experiences for pupils in Year 6 before the transition to Year 7 and 
also used an SRSD programme to embed these experiences into children’s writing. The intervention 
included professional development for primary and secondary school teachers in key elements of the 
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writing intervention SRSD, including discussion, modelling and planning. Pupils continued to receive 
the intervention, including memorable experiences, when they moved into Year 7. 


Evaluation objectives 


What is the effectiveness of the Improving Writing Quality intervention compared with ‘business as 
usual’ on the writing skills of participating children? 


Project team 


The evaluation team, jointly comprised of staff from Durham University and the University of York, was 
responsible for the design, conduct, analysis and reporting of the independent evaluation. 


Implementation team 


Calderdale Excellence Partnership, a partnership of 23 schools in West Yorkshire, was responsible for 
school recruitment and ongoing relationship with schools, informing parents and pupils, intervention 
development (including writing a detailed description of the intervention to allow others, if necessary, 
to be able to replicate the intervention in other areas), intervention training and delivery, baseline data 
collection, overseeing outcome testing and data collection. 


Ethical review 


Ethics Committees: 
e Durham University School of Education Ethics Committee; 
e York Health Sciences Research Governance Committee (by Chair's Action). 


All parents were sent a letter outlining the purpose of the study and stating that if they did not wish 
their child’s data to be sent to Durham University for analysis or to the EEF data archive then they 
were to return a form to the school to remove their child’s data from the study database. Two weeks 
were allowed to elapse after which consent was assumed. 
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Methodology 


Trial design 


The trial was designed as a pragmatic randomised controlled trial. Participating primary schools 
(feeding into participating secondary schools) were independently randomly allocated on a 1:1 ratio to 
either receive training and deliver the intervention in 2013 (the schools are labelled hereafter as being 
in the ‘intervention group’) or to continue with ‘business as usual’ in 2013 (‘control’ or ‘comparison’ 
group). Primary schools allocated to the control group were offered the intervention training at the end 
of the trial (2014). This is known as a wait-list design and is employed to increase school recruitment 
and reduce school attrition and resentful demoralisation, i.e. when participants behave in a way that 
introduces bias in response to being allocated to a control group (Torgerson & Torgerson, 2008). 
Whole school, rather than class, randomisation was used because of concerns about contamination 
between classes within schools and also to accommodate smaller schools into the study that had a 
single Year 6 class. Contamination would have occurred if schools in the control group began to follow 
the approach while the trial was still in progress, and this would have made it more difficult to assess 
the intervention’s true impact. 


In primary schools allocated to the intervention group the intervention was delivered to all pupils in 
Year 6 including those predicted to achieve below Level 4 in English at Key Stage 2. Participating 
secondary schools placed all eligible pupils (defined below) from primary intervention schools into one 
half or band of Year 7 and all eligible pupils from primary control schools into the other half or band in 
Year 7, which enabled the intervention to be delivered in Year 7 to eligible pupils from intervention 
primary schools only, minimising contamination. 


The trial was designed, conducted and reported following the CONSORT standards (Altman et al., 
2011) in order to minimise all potential threats to internal validity, such as selection bias and a range of 
post-randomisation biases (Cook and Campbell, 1969; Shadish, Cook and Campbell, 2002; Torgerson 
and Torgerson, 2008). 


Eligibility 


School recruitment and consent 


The evaluation team and Calderdale Excellence Partnership (CEP) jointly provided information 
documentation on the trial for schools (Appendix C). CEP recruited schools from West Yorkshire using 
its existing and long-standing relationship with schools in the area. School recruitment took place 
between February and March 2013. An information event for schools, run jointly by CEP and the 
evaluation team, was held on 26 February 2013. This provided an opportunity to explain both the 
intervention and the importance of the independent evaluation and what would be required of schools 
which decided to take part. Schools that wanted to take part were asked to sign an ‘agreement to 
participate form’ (Appendix D) to ensure they agreed to all trial-related procedures. 


Schools 
Primary schools were eligible to take part in the trial if (a) they were currently working with Calderdale 


Excellence Partnership and (b) they agreed to all trial procedures, including informing parents, 
provision of pupil data, randomisation and implementation of the intervention as allocated. 
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Secondary schools were eligible to take part in the trial if (a) they were currently working with the 
Calderdale Excellence Partnership and (b) they agreed to all trial procedures, including informing 
parents, provision of pupil data, continued provision of the intervention to eligible intervention pupils 
only and outcome testing. 


Pupil recruitment and consent 


Participating primary schools informed parents of all pupils in Year 6 about the study using material 
provided by the evaluation team and CEP (Appendix F). Parents had the opportunity to withdraw their 
child’s data from the analyses. Participating primary schools then shared pupil data with the evaluation 
team which included pupil name, unique pupil number (UPN), date of birth (DOB), free school meals 
(FSM), and Key Stage 2 (KS2) English teacher assessment from December 2012. Eligible pupils for 
continuation with the intervention in Year 7 were identified from this information. Pupils were eligible 
for continuation in Year 7 if they were predicted to achieve national curriculum Level 3 or insecure 
Level 4 (based on Year 6 teacher assessments). 


Participating secondary schools informed parents of all pupils in Year 7 about the study using material 
provided by the evaluation team and CEP (Appendix G). Parents had the opportunity to withdraw their 
child’s data from the analyses (opt out). Participating secondary schools then shared pupil data with 
the evaluation team. 


Pupils 


Data on all Year 6 and Year 7 pupils at participating primary schools were collected for trial purposes 
unless a parent/carer withdrew their child. Data on all year 7 pupils at participating secondary schools 
were collected for trial purposes unless a parent/carer withdrew their child. All children in Year 6 at 
primary schools allocated to the intervention group were eligible to receive the intervention. 


All pupils predicted to achieve Level 3 or an ‘insecure’ Level 4 (defined as below Level 4b) in English 
by the end of Key Stage 2 from primary schools allocated to the intervention were eligible to continue 
receiving the intervention in Year 7. (Writing assessments were based on teacher assessments 
conducted at the end of autumn term 2012.) Pupils not going to one of the three secondary schools 
who had agreed to allocate intervention and control children to different classes were not included in 
the main analysis of the programme’s impact, but were still recorded. 


Intervention 


The Improving Writing Quality intervention (delivered and supported by CEP) encouraged and funded 
primary schools to provide memorable experiences (such as a school trip to local caves and a castle) 
for all pupils in Year 6 before the transition to Year 7. Pupils who were predicted to achieve national 
curriculum Level 3 or insecure Level 4 (based on Year 6 teacher assessments) continued to receive 
the intervention when they moved into Year 7. The intervention included professional development for 
primary and secondary school teachers in key elements of the writing intervention Self-Regulated 
Strategy Development (SRSD), including discussion, modelling and planning. Initial training days were 
conducted on 18-19 March 2013 by Pooja Patel from Hill for Literacy Inc. Further training was 
conducted on 5 September for new secondary teachers who had not attended the March training. 


Following the initial training, a working group of participating teachers was established to decide how 
the strategy should be delivered in the context of the English curriculum. Guidelines were then 
prepared for all schools involved in the project. 
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General Information 


e All schools received funding to help with the provision of memorable experiences. 
e The memorable experiences were used as the focus for pupils’ writing. 


e All schools in the intervention group were required to use the main elements of the SRSD 
writing programme (listed below) in the teaching of writing. 


e Secondary schools received details of all the pupils involved in the programme at primary level 
and these pupils were then part of the programme at secondary level. 


e All pupils at secondary schools could take part in any funded memorable experiences but only 
those pupils in English sets which contained pupils who were part of the project received 
tuition in the SRSD method for developing writing. 


e The SRSD method of developing writing was used throughout the autumn term with the 
intervention groups in Year 7 in each of the participating secondary schools. 


e All pupils working at NC Level 3 and above in Year 7 took the GL Writing test in December 
2013. 


e Arrangements were made for visits to schools to monitor and support the implementation of 
the programme at both primary and secondary level. 


Timescales — dates and times for implementation 
e Primary teachers were able to use elements of SRSD immediately after the training or could 
delay using the strategy until after SATs; most chose the latter. 


e SRSD had to be taught explicitly as part of Literacy lessons and further developed through 
topic work. 


e Secondary teachers used the SRSD approach with English sets which contained pupils in the 
intervention group for the teaching of writing in the autumn term. 


Delivering the programme 


Teachers delivered SRSD in the order outlined in the manual but were allowed to combine lessons 
rather than stick rigidly to the prescribed model. 


When teaching a specific genre all teachers were asked to include the following elements: 


e discussion about the genre 


e pre assessment — carried out by class teachers before starting the SRSD programme on any 
genre 


e mnemonics — the mnemonic recommended for use was IPEELL: I= Introductory paragraph; 
P= Points; E = Examples/elaboration; E=End; L=Links (connectives, openers); L = Language 
(wow words, genre specific vocabulary, punctuation) 


e graphic organisers -— all schools were asked to use the same graphic organiser (planning 
scaffold) 


e _ self-scoring and graphing — self-scoring systems were required to be used based on examples 
in the manual, with the results always graphed 
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e ‘self-talk’ — to be used at all stages (the American practice of ‘self-talk’ was Anglicised using 
the terms ‘positive talk’ and ‘motivational messages’) 


e peer scoring 


e = final assessment. 


Monitoring and support 


e Monitoring visits took place in June and July in primary schools. 
e Monitoring visits took place in October and November in secondary schools. 


e Support was available from April 2013 for all schools. 


Liaison with secondary colleagues 


e Some secondary colleagues took the opportunity to visit primary schools to see SRSD in 
action and to discuss what had been covered so that teachers could build on existing 
experiences in the autumn term. 


Development of toolkit 


e After the project is completed, teachers involved will meet to discuss the implementation of the 
programme and to put forward ideas for materials that could be included in a toolkit that will 
help any other schools that wish to use the strategy. 


Outcomes 


The Progress in English (PiE) 11: Second Edition Long Form (LF) Test, GL Assessment, was the main 
literacy outcome measure. The test includes both narrative and non-narrative exercises and assesses 
both reading and writing skills including areas such as spelling, grammar and comprehension. The 
Progress in English Test was the only test available to the evaluation team (in order to comply with 
EEF testing policy) which included a writing component. 


All Year 7 pupils in participating secondary schools undertook the test in December 2013 with the 
exception of pupils identified as below Level 3 (for whom the test is not suitable). Schools were 
instructed to deliver the tests under ‘exam’ conditions. The whole class sat the test at the same time. 
Tests were marked by GL Assessment ‘blind to allocation’ (i.e. markers did not know whether test 
papers were from intervention or control pupils). 


Primary outcome 


The primary outcome was extended writing score which refers to the combined raw score on the two 
extended writing tasks (Exercises 5 and 6) from the PiE test. Exercise 5 has a maximum of 20 marks 
and involves writing a persuasive letter. Exercise 6 has a maximum of 12 marks and assesses 
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informative writing. Overall, the extended writing task score can be in the range 0 to 32, with a higher 
score representing higher attainment. 


Secondary outcome 


Reading score, the combined raw score on the reading tasks (Exercises 3, 4, 3x and 4x), was used as 
the secondary outcome. Exercise 3 (comprising Exercises 3 and 3x) has a maximum total of 19 marks 
and assesses reading comprehension of a narrative. Exercise 4 (comprising exercises 4 and 4x) has a 
maximum of 13 marks and assesses non-narrative reading comprehension. Overall, reading score 
can range between 0 and 32, with a higher score representing higher attainment. 


Spelling and grammar score, the combined raw score on the spelling and grammar tasks (Exercises 1 
and 2) was chosen as a further secondary outcome. Exercise 1 has a maximum of 10 marks and 
assesses spelling. Exercise 2 also has a maximum of 10 and assesses grammar. This means the 
spelling and grammar score combined can range from 0 to 20, with a higher score representing higher 
attainment. 


Sample size 


The focus of this evaluation was on children who were performing at either national curriculum Level 3 
or insecure Level 4 in English and hence the sample size calculation was based upon this population 
of children. 


Prior to recruitment it was anticipated that approximately 24 primary schools would be involved in this 
study. The average Year 6 cohort in Calderdale was estimated at 45 pupils and it was predicted that 
between a quarter and a third of pupils would meet the eligibility criteria. This meant that, on average, 
there would be between 11 and 15 eligible pupils per primary school. Assuming, conservatively, 12 
pupils per school meant that around 288 pupils would be eligible for the trial (144 per group). 
Assuming an intra-cluster correlation coefficient of 0.19 (from ECC randomised trial, Torgerson et al., 
2011) would give a design effect of 3.09. Dividing the number of pupils in the trial (288) by this design 
effect gives an ‘effective sample size’ of approximately 94 pupils. However, we also assumed a pre- 
and post-test correlation of 0.70 which inflates the effective sample size to 192 pupils. After allowing 
for an estimated attrition rate of 10% this leads to an effective sample size of 172 pupils. This would 
give 80% power to show a difference of 0.43 standard deviations in writing score between the two 
groups providing a difference of this magnitude exists. 


Randomisation 


Primary schools were allocated to either intervention or control on a 1:1 ratio using deterministic 
minimisation. Minimisation is a technique that ensures balance between the groups by using an 
arithmetical algorithm. This process was conducted by the independent evaluation statistician using 
minimPy (http://sourceforge.net/projects/minimpy/). The number of pupils on roll, percentage of pupils 
in minority ethnic groups and percentage of pupils eligible for free school meals (FSM) were used as 
minimisation factors. The first two of these factors used three levels while FSM used two levels. The 
algorithm calculates the balance on specified variables after each individual has been allocated such 
that the next allocated individual minimises any chance imbalance between the groups (Torgerson & 
Torgerson, 2008). 
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The evaluation team provided allocation information to the implementation team for them to 
disseminate this to the schools. Due to an administrative error within the evaluation team, there was a 
discrepancy between the minimisation output and the allocation information which was provided to 
CEP in relation to two schools. This meant that one school was informed that they had been allocated 
to the control group when in fact the minimisation had assigned them to receive the intervention and 
that another was informed they had been allocated to the intervention arm when in fact the 
minimisation had assigned them to the control arm. As this administrative error occurred before the 
delivery partners or schools were informed of allocations, and therefore was an independent error, no 
bias should be introduced. The schools, therefore, were analysed according to the allocations 
received by the schools. 


Analysis 


Analysis was conducted in Stata® version 13 (Stata Corporation, College Station, Texas, USA) using 
the principles of ‘intention-to-treat’, meaning that all schools and pupils were analysed in the group 
they were randomised to irrespective of whether or not they actually received the intervention. 


Statistical significance was assessed at the 5% level unless otherwise stated. Regression based 
methods of analysis were used and 95% confidence intervals are provided as appropriate. 


Effect sizes were calculated and are presented alongside 95% confidence intervals. Effect size is 
defined as: 


A = Bintervention 
Os 


where Bintervention is the difference in mean score between the intervention and control groups and oz is 
the residual standard deviation. The residual standard deviation was used rather than the more usual 
pre-test standard deviation as there was no equivalent pre-test value. Numerical values used to 
calculate the effect sizes for each analysis can be found in Appendix Q. 


The test and outcomes were examined for ceiling or floor effects using summary statistics and 
graphical representations. 


Estimates of the intra-cluster correlation coefficients (ICCs) for secondary school class and primary 
school in terms of both primary outcome and raw score are provided. No estimates are presented for 
secondary schools due to the small number of schools involved. The correlation between predicted 
KS2 Level and both raw score and primary outcome is also explored. 


Primary analysis 


The primary objective of this study was to investigate the effectiveness of the intervention on the 
writing skills of children meeting the inclusion criteria described previously: Year 6 at the beginning of 
the study and predicted to achieve Level 3 or an insecure Level 4 in English by the end of Key Stage 
2. The difference in extended writing scores between pupils in the intervention and control groups was 
compared using a cross-classified multilevel model to account for potential clustering. Such a model is 
required as the data are not strictly hierarchical. Adjustment was made for predicted KS2 score, 
gender, EAL status, FSM status and month of birth. Secondary school class and primary school were 
included in the model as random effects with an artificial super cluster created at the secondary school 
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class level. Due to the small number of secondary schools involved in this trial, secondary school was 
not included as a random effect in the primary analysis. 


Secondary analyses 


An analogous approach to the primary analysis was used to examine differences between the 
intervention and control groups in terms of the secondary outcomes of reading, spelling and grammar 
for pupils meeting the inclusion criteria. 


To assess the impact on pupils who were not eligible to receive the intervention and thus only 
receiving the intervention in Year 6, a similar analytical approach to that used in the primary analysis 
was applied three times including pupils predicted to achieve a Level 4a or above. Each repetition of 
this analysis looked at differences in extended writing score, reading score, and spelling,and grammar 
score respectively. 


A similar analytical approach was used to assess the effectiveness of the intervention on writing skills 
for all children at Level 3 and above who received the intervention in either just Year 6 (those 
ineligible), or Years 6 and Year 7 (those eligible). 


Regression models were also used to compare differences between the intervention and control 
groups with respect to the secondary outcomes of reading, spelling and grammar for all children at 
Level 3 and above. 


Sub-group analysis 


The effect of the intervention in terms of extended writing score was also analysed for the sub-group 
of pupils eligible for free school meals (FSM) through the inclusion of an interaction between FSM 
status and allocation in a further repetition of the primary analysis. This sub-group was chosen by the 
funder a priori before the study started and is a subgroup of interest for all EEF projects. Statistical 
significance was set at the 10% level as this trial was not powered to detect interactions. 


As required by the funding body, an analogous regression to that used in the primary analysis was 
used to examine the effect of the intervention on pupils eligible for FSM, excluding all those not eligible 
for FSM. 


Sensitivity analyses 


The primary analysis was repeated including secondary school as a random effect. 


Process evaluation methodology 


A ‘light touch’ process evaluation was conducted, primarily to assess fidelity and potential 
contamination. During CEP’s observations of primary schools allocated to the intervention group a 
score of ‘fidelity to intervention’ was recorded based on two questions: (1) Are the teachers delivering 
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SRSD intervention in line with the training received? (fully, partially, not at all) and (2) Do participating 
pupils have the opportunity to practise SRSD techniques? (fully, partially, not at all). 


CEP also conducted a brief survey of all participating primary schools (both intervention and control) 
to collect information on the memorable experiences conducted by schools, and (from intervention 
schools only) to obtain views on the impact of the SRSD intervention. 


In addition: the evaluation team had on-going discussions and communication with the delivery partner 
CEP throughout the trial period, a primary school classroom observation, and an informal interview 
with a primary school teacher who had delivered the intervention was conducted. 
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Participants 
School recruitment 


CEP approached 24 primary schools to consider taking part in the trial (Table 1). All schools were 
members of CEP and were positive about taking part; however one school was excluded because it 
felt unable to deliver the intervention in 2013 if allocated to the intervention group due to being in 
special measures. 


CEP approached three secondary schools, all of whom were willing to implement all necessary trial 
procedures. 


Table 1. School recruitment 


Primary 24 23 96% 
Secondary 3 3 100% 
Pupil recruitment 


All 23 primary schools sent opt-out letters to parents of all children in Year 6. In total 865 letters were 
sent and 23 (2.7%) opt-out forms were returned. 


All three secondary schools sent opt-out letters to parents of all children in Year 7. In total nine opt-out 
forms were returned: two of these pupils had also opted-out at Year 6 and there were seven new opt- 
outs. 


Participant flow 


Figure 1 shows the flow of participants who were involved in Year 6 through the study. There were 
842 pupils involved in the study at the point of randomisation: 787 of these were predicted to achieve 
Level 3 or above in KS2 English and hence were eligible for testing. There were 432 pupils (51.3% of 
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842) who were predicted to achieve a Level 3 or insecure Level 4 in KS2 English and hence were 
eligible for inclusion in the primary analysis. 


Twelve schools (containing 430 participating pupils) were randomised to the control condition and 11 
schools (containing 412 participating pupils) were randomised to receive the intervention in 2013. 
There were 209 pupils in control schools and 223 pupils in intervention schools who were eligible for 
inclusion in the primary analysis. 


When the pupils made the transition to secondary school, 193 of the 430 pupils at control schools did 
not attend participating secondary school; 79 of these 193 pupils would have been eligible for 
inclusion in the primary analysis, had they attended one of the three secondary schools. These 79 
pupils did not attend the three designated secondary schools for a variety of reasons such as parental 
choice or moving out of the area. This left 237 pupils from control primary schools participating in the 
trial in Year 7, of whom 130 were eligible for inclusion in the primary analysis. There were 131 pupils 
from intervention primary schools who did not attend a participating secondary school; 69 of these 
pupils would have been eligible for inclusion in the primary analysis. This left 281 pupils from 
intervention primary schools participating in the trial in Year 7, of whom 154 were eligible for inclusion 
in the primary analysis. 


After the exclusion of pupils without the primary outcome of extended writing score, the primary 
analysis was conducted on 142 intervention pupils and 119 control pupils. 


There were 518 pupils who attended both a participating primary and secondary school; of these 480 
were eligible for testing. Only 7.7% (37 out of 480) of these pupils did not complete all the writing 
exercises on the test and hence did not have a primary outcome. Slightly more males than females did 
not complete all the writing components (23 compared to 14) and more pupils who were not eligible for 
FSM had a missing value than those who were eligible for FSM. Just under half of the 37 pupils with 
no primary outcome had English as an additional language, and these had predicted levels ranging 
from Level 3c to Level 5b. Nearly half of these pupils (18 of 37) were missing secondary outcome 
measures of reading and spelling in addition to the primary outcome. 
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Figure 1. CONSORT diagram 


Primary schools recruited 


(N= 24) 


Pupils: n= 865 
School excluded (unable to implement 
intervention in 2013) (N=1) 


Opt-outs (n= 23) 


Primary school randomised (N= 23) 
Pupils: n= 842 
Eligible for testing: n= 787 
@ Eligible for inclusion in primary analysis: n=432 


Randomised to control (N= 12) Randomised to intervention (N = 11) 
Pupils: n= 430 Pupils: n= 412 
Eligible for testing: n= 407 © Eligible for testing: n= 380 


© Eligible for inclusion in primary analysis: n=209 @ Eligible for inclusion in primary analysis: n=223 


¥6 intervention period 


Did not attend a participating secondary 
school 


Did not attend a participating secondary 


{All 412 pupils at intervention school receive school 


intervention) 


Pupils: n= 193 (of whom 79 eligible 
for inclusion in primary analy sis} 


Pupils: n= 131 (of whom 69 eligible 
for inclusion in primary analy sis) 


Attended a participating secondary school 
Pupils: n= 237 
@ Eligible for testing: n= 224 
@ Eligible for inclusion in primary analysis: n=130 


Attended a participating secondary school 
Pupils: n= 281 
@ Eligible for testing: n= 256 
@ Eligible for inclusion in primary analysis: n=154 


¥? intervention period 


Post intervention testing 


Primary analysis 


Primary analysis 


Eligible for inclusion: n=154 
e Included: n= 142 


Eligible for inclusion: n=130 
e Included: n= 119 


@ Excluded (missing primary outcome): n=11 e Excluded (missing primary outcome): n=12 


School characteristics 


Table 2 shows summary statistics of school-level characteristics of the 23 primary schools which were 
involved in the trial. The mean school size was just over 300 pupils with a range between 82 and 581. 
Just over one third of pupils on roll had been eligible to receive free school meals at some point in the 
previous 6 years and just under one third of pupils were from minority ethnic groups: this is slightly 
higher than the national average in January 2013 which was 28.5% (Department for Education, 2013). 
The proportion of FSM pupils was similar to the national average which is 19.2% for all maintained 
nursery and primary schools. All characteristics were fairly similar between the trial arms with the 
possible exception of minority ethnic groups where the control schools tended to have a higher 
proportion of such pupils than the intervention schools. 
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Intervention Control Overall 
n=11 n=12 n=23 
Number of pupils on roll 
Mean (SD) 312 (165.6) 308.2 (146.7) 310 (152.4) 


Median (min, max) 


231 (82, 581) 


283.5 (127, 531) 


283 (82, 581) 


% eligible for FSM Ever 6 


Mean (SD) 39.4 (18.5) 33.5 (15.0) 36.3 (16.7) 
Median (min, max) 40 (6.1, 68.1) 34.5 (8.9, 61) 37.2 (6.1, 68.1) 
% pupils from minority ethnic groups 

Mean (SD) 22.5 (33.5) 39.0 (40.0) 31.1 (37.2) 
Median (min, max) 7.5 (1, 96) 25.1 (2.4, 99.2) 10.2 (1, 99.2) 
% pupils supported by school action 

Mean (SD) 13.2 (5.0) 13.5 (4.9) 13.4 (4.8) 
Median (min, max) 12.6 (5.7, 20) 12.2 (6.7, 21.5) 12.6 (5.7, 21.5) 
% stability 

Mean (SD) 81.1 (6.0) 83.6 (8.5) 82.4 (7.4) 


Median (min, max) 


80.9 (73.5, 91) 


85.3 (61, 95.2) 


83.7 (61, 95.2) 


Pupil characteristics 


Baseline characteristics for all those involved in the trial in Year 6 are presented in Table 3. At the 
point of baseline data collection and randomisation there were 842 pupils attending participating 
primary schools. Data collected in Year 7 suggested a further nine pupils attended participating 
primary schools, however no match within the Year 6 was available for any of these individuals. It is 
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possible that this discrepancy could be due to a misreported primary school in the Year 7 data or due 
to the fact that the pupils did not attend a participating primary school at the time of baseline data 
collection in December 2013. These pupils are therefore excluded from analyses and are not included 
in Year 6 figures. Proportions of pupils at each predicted level were fairly similar between trial arms; 
however the proportion of FSM pupils was slightly higher in the intervention group at 32.5% compared 
to 23.7% of the control group. 


Table 3. Pupil level characteristics for all pupils 


Intervention Control 
n=412 n = 430 

FSM 
Eligible 134 (32.5) 102 (23.7) 
Not eligible 278 (67.5) 328 (76.3) 
Missing 0 (0.0) 0 (0.0) 
Predicted KS2 writing level 
Level 2 or below 32 (7.8) 23 (5.4) 
Level 3c 14 (3.4) 17 (4.0) 
Level 3b 29 (7.0) 29 (6.8) 
Level 3a 27 (6.6) 19 (4.4) 
Level 4c 77 (18.7) 69 (16.1) 
Level 4b 76 (18.4) 75 (17.5) 
Level 4a 56 (13.6) 62 (14.5) 
Level 5 or above 100 (24.3) 135 (31.4) 
Missing 1 (0.2) 1 (0.2) 


At baseline there were 432 pupils eligible to receive the intervention in both Year 6 and Year 7 — 209 
control pupils and 223 intervention pupils. Of these 432 pupils, 148 (34.3%) did not attend a 
participating secondary school. Baseline characteristics for the 284 pupils predicted to achieve 
between a Level 3c and a Level 4b who also attended a participating secondary school are presented 
in Table 4 both as randomised and as analysed. The KS2 predicted level, which is the strongest 
predictor of the primary outcome, is closely balanced in the analysed groups (i.e. 0.66 standard 
deviations difference). 
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Table 4. Pupil-level characteristics for pupils predicted a Level 3 or insecure Level 4 and 
attending a participating secondary school 


Intervention Control Intervention Control 
Number of eligible pupils in Year 7 n= 154 n= 130 n= 142 n=119 
Gender t 
Male 84 (54.6) 78 (60.0) 77 (54.2) 70 (58.8) 
Female 70 (45.5) 52 (40.0) 65 (45.8) 49 (41.2) 
Missing 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 
FSM 
Eligible 57 (37.0) 38 (29.2) 50 (35.2) 36 (30.3) 
Not eligible 97 (63.0) 92 (70.8) 92 (64.8) 83 (69.8) 
Missing 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 
English as an additional languaget 
EAL 64 (41.6) 77 (59.2) 60 (42.3) 69 (58.0) 
Non-EAL 90 (58.4) 53 (40.8) 82 (57.8) 50 (42.0) 
Missing 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 
Predicted KS2 writing level 
Level 3c 9 (5.8) 6 (4.6) 8 (5.6) 4 (3.4) 
Level 3b 23 (13.9) 25 (19.2) 20 (14.1) 20 (16.8) 
Level 3a 18 (11.7) 9 (6.9) 15 (10.6) 8 (6.7) 
Level 4c 49 (31.8) 43 (33.1) 47 (33.1) 40 (33.6) 
Level 4b 55 (35.7) 47 (36.2) 52 (36.6) 47 (39.5) 
Missing 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 
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Mean predicted KS2 writing levels n=142 n=119 


3.81 (1.23) 3.89 (1.20) 


Month of birtht 


Sept — Nov 38 (24.7) 27 (20.8) 35 (24.6) 24 (20.2) 
Dec — Feb 41 (26.6) 30 (23.1) 37 (26.1) 29 (24.4) 
Mar — May 40 (26.0) 34 (26.2) 38 (26.8) 31 (26.1) 
Jun — Aug 35 (22.7) 39 (30.0) 32 (22.5) 35 (29.4) 
Missing 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 


t Only collected in Year 7, however these are time constant variables and as such are still presented. 


Outcomes and analysis 


Of the 23 primary schools, eight sent pupils to only one secondary school, seven sent pupils to two 
different secondary schools and another seven sent pupils to all of the secondary schools involved in 
the trial. One school did not send any pupils to a participating secondary school. 


Of the three secondary schools, one was fed into by 10 participating primary schools, another had 14 
feeder primaries and the last received pupils from 19 of the participating schools. 


There were 28 Year 7 classes spread across the three secondary schools. One secondary school 
class had no individuals from a participating primary school, the other 27 classes contained pupils 
from between two and 23 participating primary schools. 


The post-test and outcomes were assessed for ceiling or floor effects using histograms and summary 
statistics. Ceiling effects occur if a test is too easy (e.g. lots of pupils achieve a perfect score), while 
floor effects occur if a test too hard (e.g. lots of pupils score zero). Both types of effect can unfairly 
reduce the apparent impact of the intervention. No evidence of either effect was found (details in 
Appendix O). 


Intra-cluster correlation coefficients (ICCs) were estimated (Table 5). The secondary class ICC based 
on the primary analysis (0.10) is slightly smaller than the one that was used for the sample size 
calculation estimate (0.19) which was derived from a maths study. All primary school ICCs were fairly 
small. However, as post test data were not available for pupils who did not attend a participating 
secondary school, this is based on a sub-group of those who were involved in the trial in Year 6. 


The correlation between primary outcome (the extended writing score) and the predicted KS2 level for 
those eligible for inclusion in the primary analysis was also lower than expected (Spearman’s Rho 
0.35). The correlation between overall test raw score and predicted KS2 level based on all participants 
was 0.64. 
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Table 5. Estimated intra-cluster correlation coefficients (ICCs). 


Total raw score 360 0.51 0.03 


Primary outcome 499 0.40 0.04 
(extended writing score) 


Primary outcome based 261 0.10 0.08 
on primary analysis 


Raw, unadjusted mean post-test scores are presented in Table 6 by trial arm for those eligible for 
inclusion in the primary analysis (i.e. attended both a participating primary school and a participating 
secondary school and predicted Level 3 or insecure Level 4 in Year 6). In terms of the primary 
outcome of extended writing, the mean score was higher in the intervention group than in the control 
groups at 21.9 out of 32 marks (SD 4.39) in the intervention group compared to 19.4 marks (SD 5.32) 
in the control group. The proportion of those completing the extended writing questions was slightly 
higher in the intervention arm than in the control arm at 63.7% compared to 56.9% of the control arm. 
In general the completion rate of the primary outcome was high at 91.9% of those eligible for inclusion 
in analysis who attended a participating secondary school (261 out of 284). The majority of those 
completely missing answers to all the extended writing questions (148 of 163, 90.8%) did not attend a 
participating secondary school and hence were not tested. Mean reading score and mean spelling and 
grammar score were similar between the arms. The proportion of pupils completing the reading 
outcome was slightly lower in the intervention group than in the control group. The opposite was true 
for the spelling outcome, where a slightly higher proportion of intervention pupils completed than 
control pupils. For both secondary outcomes, the majority of pupils completely missing outcome 
scores did not attend a participating primary school. 
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Table 6. Unadjusted average scores for the intervention and control groups for those 


eligible for inclusion in the primary analysis 


Eligible for inclusion in primary analysis in Year 6 n= 223 n= 209 n= 432 
Eligible for inclusion in primary analysis and n= 154 n= 130 n= 284 
attended a participating secondary school 

Primary outcome 

Extended writing score n= 142 n=119 n= 261 
Mean (SD) 21.9 (4.39) 19.4 (5.32) 20.8 (4.97) 
Med (Min, Max) 22 (8, 31) 19 (8, 32) 21 (8, 32) 
Completely missing & not attending participating 69 (30.9) 79 (37.8) 148 (34.3) 
secondary school (%) 

Completely missing & attending participating 10 (12.7) 5 (6.0) 15 (3.5) 
secondary school (%) 

Partially missing (%) 2 (0.9) 6 (2.9) 8 (1.9) 
Complete (%) 142 (63.7) 119 (56.9) 261 (60.4) 
Secondary outcomes 

Reading score n= 108 n = 88 n= 196 
Mean (SD) 14.5 (4.38) 14.3 (4.79) 14.4 (4.56) 
Med (Min, Max) 14 (5, 26) 14 (3, 26) 14 (3, 26) 
Completely missing & not attending participating 69 (30.9) 79 (37.8) 148 (34.3) 
secondary school (%) 

Completely missing & attending participating 9 (4.0) 6 (2.9) 15 (3.5) 
secondary school (%) 

Partially missing (%) 37 (16.6) 36 (17.2) 73 (16.9) 
Complete (%) 108 (48.4) 88 (42.1) 196 (45.4) 
Spelling and grammar score n= 142 n=112 n= 254 
Mean (SD) 8.1 (4.17) 8.8 (4.80) 8.4 (4.46) 
Med (Min, Max) 8 (0, 17) 9 (0, 20) 8 (0, 20) 
Completely missing & not attending participating 69 (30.9) 79 (37.8) 148 (34.3) 
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secondary school (%) 


Completely missing & attending participating 8 (3.6) 8 (3.8) 16 (3.7) 
secondary school (%) 

Partially missing (%) 4 (1.8) 10 (4.8) 14 (3.2) 
Complete (%) 142 (63.7) 112 (53.6) 254 (58.8) 
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Primary analysis — impact of approach on extended writing of pupils between Level 3c 
and 4b 


The primary analysis was adjusted for baseline-predicted KS2 writing level, gender, FSM status, EAL 
status and month of birth. After exclusion for missing data relating to any of these variables or the 
response, analysis was conducted on 261 pupils: 142 from the intervention group and 119 from the 
control group. There was evidence of a difference in extended writing score between the allocated 
groups, with a significant increase of 2.53 marks for those in the intervention group when compared 
with those in the control group (p=0.002, 95% Cl: 0.90 to 4.16). This relates to an effect size of 0.74 
(95% Cl: 0.26 to 1.22). Full details of coefficients for all covariates can be found in Appendix P. 


Secondary analyses 


The primary analysis was repeated using reading score as the response. After exclusion for missing 
data relating to any covariates or the response, analysis was conducted on 196 pupils: 108 from the 
intervention group and 88 from the control group. There was no evidence of a difference in reading 
scores between the allocated groups, with a non-significant decrease of 0.31 marks for those in the 
intervention group when compared with those in the control group (p=0.72, 95% Cl: -2.02 to 1.41). 
This relates to an effect size of -0.09 (95% Cl: -0.59 to 0.41). 


The primary analysis was also repeated using spelling and grammar score as the response. After 
exclusion for missing data relating to any covariates or the response, analysis was conducted on 254 
pupils: 142 from the intervention group and 112 from the control group. There was no evidence of a 
difference in spelling and grammar score between the allocated groups, with a non-significant 
decrease of 0.44 marks for those in the intervention group when compared with those in the control 
group (p=0.50, 95% Cl: -1.72 to 0.84). This relates to an effect size of -0.13 (95% Cl: -0.51 to 0.25). 


Pupils at Level 4a and above 


To assess the impact on the writing skills of pupils who were not eligible to receive the intervention 
and thus only received the intervention in Year 6, the primary analysis was repeated using only pupils 
at Level 4a and above. After exclusion for missing data relating to any of covariates or the response, 
analysis was conducted on 181 pupils; 94 from the intervention group and 87 from the control group. 
There was no evidence of a difference in extended writing score between the allocated groups, with a 
non-significant increase of less than 0.01 marks for those in the intervention group when compared 
with those in the control group (p=1.00, 95% Cl: -1.38 to 1.38). This relates to an effect size of 
approximately 0.00 (95% Cl: -0.45 to 0.45). This could be due to a number of factors, for example the 
fact that these pupils did not receive the full intervention. 


This analysis was repeated using the same individuals (those at Level 4a or above) but with reading 
score as the response variable. After exclusion for missing data relating to any of the covariates or the 
response, analysis was conducted on 148 pupils; 81 from the intervention group and 67 from the 
control group. There was no evidence of a difference in reading score between the allocated groups, 
with a non-significant decrease of 0.39 marks for those in the intervention group when compared with 
those in the control group (p=0.59, 95% Cl: -1.80 to 1.02). This relates to an effect size of -0.12 (95% 
Cl: -0.58 to 0.33). 


This analysis was repeated on the same individuals again (those at Level 4a or above) but with 
spelling and grammar score as the response variable. After exclusion for missing data relating to any 
of the covariates or the response, analysis was conducted on 182 pupils: 95 from the intervention 
group and 87 from the control group. There was no evidence of a statistically significant difference in 
spelling and grammar score between the allocated groups, with a non-significant increase of 0.10 
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marks for those in the intervention group when compared with those in the control group (p=0.89, 95% 
Cl: -1.43 to 1.64). This relates to an effect size of 0.04 (95% Cl: -0.49 to 0.56). 


Sub-group analysis 


A pre-specified subgroup analysis was conducted. The effect of the intervention on pupils eligible for 
FSM was assessed through the inclusion of an interaction term in a repetition of the primary analysis. 
Statistical significance was assessed at the 10% level. There was no evidence of a statistically 
significant interaction between allocated group and FSM status (p=0.69) suggesting the intervention 
did not have a differential effect dependent on FSM status. 


As required by the funding body, the primary analysis was repeated in the subgroup of pupils eligible 
for FSM. After exclusion for missing data relating to any of these variables or the response and for 
pupils not eligible to receive FSM, analysis was conducted on 86 pupils; 50 from the intervention 
group and 36 from the control group. There was evidence of a difference, which reflects the main 
analysis, in extended writing score between the allocated groups, with a significant increase of 3.34 
marks for those in the intervention group when compared with those in the control group (p=0.02, 95% 
Cl: 0.55 to 6.13). This relates to an effect size of 1.60 (95% Cl: 0.21 to 2.98). Note, however, that 
although the effect size for this subgroup appears to be much larger than the overall effect size, the 
test for interaction, described above, showed no evidence that this apparent difference was caused by 
anything other than chance. 


Sensitivity analysis 


When the primary analysis was repeated also accounting for clustering at the secondary school level, 
results were consistent with those obtained in the primary analysis. 


Cost 
The costs to schools of repeating the programme is estimated at: 


- one day’s training — venue, equipment, refreshments and training materials (£900) 
- SRSD trainers, with one day’s preparation (£900 per trainer) 
- memorable experiences per class (£1,500). 


The following assumptions are built into the calculations for implementation by other schools: 


- _SRSD training delivered to 30 teachers 
- SRSD delivered to a class of 30 pupils. 


Basic costs would then be: £60 per teacher for the initial training and £50 per pupil for memorable 
experiences i.e. £1,560 per class. 


These costs could be reduced significantly depending on the cost of the memorable experiences 
chosen; they would be increased if supply cover for the teacher training was required. 


Additional resources required would be minimal as the strategies can be delivered as an integral part 
of literacy/English teaching. The project team is developing a toolkit by 2015 (teacher manual, DVDs 
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etc.) which, if reproduced commercially, is estimated will cost in the region of £50 to £100. A school 
would only need to buy one toolkit. 


Costs in future years would relate to the cost of the chosen memorable experiences and any further 
training required for teachers new to the school. 


Process evaluation 


Only two of the intervention schools opted to implement the intervention before the Standard 
Assessment Tests (SATs) in May (as they did not wish to disrupt children’s progress towards the 
SATs). The implication of this is that in most primary schools the intervention was only delivered over 
approximately a six-week period. Generally the feeling, reported by CEP, was that most schools would 
have preferred to have started using the intervention earlier, with many teachers commenting on the 
positive impact it had had on children within their class. Most of the intervention primary schools also 
reported to CEP that they intended to use the intervention from September with the following year’s 
Year 6 and Year 7 children and to introduce it into other year groups. 


Classroom observation 


One classroom observation in total was conducted, and therefore any findings and conclusions should 
be interpreted with extreme caution. This session was delivered especially for the evaluation team (at 
the end of summer term). The session clearly demonstrated many of the essential elements of the 
SRSD intervention that had been outlined to schools by CEP related to planning to write. It was clear 
that the session was the first of two, and it covered all the elements related to planning and preparing 
to write. The session did not involve the children actually writing, or conducting any self-scoring or 
graphing: this was due to be covered in a second session. 


Informal teacher interview 


An informal discussion was held with one teacher who had been delivering the intervention, along with 
a member of CEP. In her view, the training received was too ‘American’ and ‘not the way things would 
be done here’, especially the ‘self-talk’ and motivational elements. She suggested that the training 
could have been condensed into one day rather than two. She had not accessed any of the additional 
resources suggested and also noted that she had not made use of the manual provided at the training 
day, except when, as a school, they designed their own scaffold. She noted that CEP had given them 
flexibility and that they had gone on to adapt and use the intervention as they felt suited their school. 
Overall the teacher was extremely positive about the intervention (although noted this was as adapted 
and used by CEP rather than the way they were trained). She felt the intervention had been very 
beneficial and intended to continue using it the following year. She also felt that the children had 
internalised the techniques and that this was demonstrated in the ‘cold writing tasks’ where children 
were given no guidance but used the techniques taught of their own accord. 


The teacher noted that at the beginning some children were spending more time on the mathematical 
elements (graphs), and as a school they knew they needed to alter this as it clearly wasn’t the purpose 
of the intervention. 


The teacher reported that children enjoyed the scoring (this school used self-scoring, peer-scoring and 
teacher scoring) and the graphing element of the intervention as it allowed them to see how they could 
make progress, and increased their confidence. 
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This school did not start using the intervention until after SATs and reported that there had been too 
little time after SATs, but that they intended to use the intervention from the following September with 
the new Year 6. 


The teacher also reported that she did not feel that the children’s writing skills improved solely due to 
writing about a memorable experience, as at the beginning of the intervention period they had 
conducted a ‘cold writing task’ based on a memorable experience, because they were interested to 
know for themselves if the SRSD intervention would make a difference above and beyond writing 
about a memorable experience. The teacher reported that, in her view, that the SRSD intervention 
was the element that made the difference not just writing about memorable experiences. 


This school conducted their own process evaluation of the intervention by asking participating children 
their views on the intervention. A large majority of the children noted that they had found the self- 
scoring and graphing very important and helpful as it allowed them to see what elements of their 
writing they needed to improve. It also increased their confidence when they could visually see they 
were making progress. 


Discussions with Calderdale Excellence Partnership 


From ongoing discussions with CEP throughout the trial period, it was clear that a process of 
‘intervention development occurred over the trial period and is ongoing. CEP gave the primary 
schools flexibility to develop and implement some components of the intervention as each saw fit 
following the training received in March 2013. CEP also noted the ‘American-ness’ of the intervention 
and felt it needed to be to be Anglicised for the English classroom context. This was especially true of 
the ’self-talk’ element which CEP reported many of the primary schools had reframed into a more 
British approach. 


CEP delivered their own refresher training to secondary schools in September, and incorporated 
‘lessons learnt’ and adaptations following the intervention period in primary schools. CEP have set up 
a working group to discuss how the intervention should be modified and adapted for implementation 
next year. 


Implementation 


One teacher at one secondary school received training but taught a control class; at another 
secondary school, two teachers taught intervention classes without having had the initial SRSD 
training. 
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Conclusion 


Conclusion 


The results of this large cluster randomised controlled trial are in line with a systematic review of 
previous, largely American, trials showing SRSD is an effective strategy for teaching writing skills. 
Though the finding is consistent with existing research, the positive effect size for the primary outcome 
(in excess of 0.70 standard deviations) was nonetheless remarkably large. Although there appeared to 
be an even larger difference within the FSM subgroup, this difference between FSM and non-FSM 
pupils was not close to being statistically significant. Consequently, there is a high likelihood that the 
difference is simply due to chance. 


Strengths of the evaluation 


In the design and conduct of our study we used best practice as defined by the CONSORT guidelines 
for randomised controlled trials (Altman et al., 2011). Importantly, we used independent concealed 
allocation to ensure that the clusters were allocated without the possibility of bias. We used the 
principles of intention-to-treat by including all consenting eligible classes, teachers and children in the 
final analysis. We pre-specified our main outcome and wrote a statistical analysis plan before we 
observed the data. We also used an independent company to mark the test papers, blind to the 
allocated group. 


Limitations of the evaluation 


The trial was only undertaken within a narrow geographical area of the UK, however about 30% of 
pupils were eligible for FSM and 30% were from ethnic minorities, suggesting that the results are 
applicable across different sub-groups within the population. From the process evaluation there is 
evidence that teachers adapted the SRSD intervention into an English context as they felt that it was 
too ‘American’; therefore the description of the intervention may not be actually what was implemented 
within the typical classroom. In addition, we cannot disentangle the effectiveness of the individual 
components of the intervention (e.g. the training or the implementation of the intervention with pupils in 
primary or secondary schools). 


Interpretation 


This intervention appears to be highly effective at improving children’s writing skills despite imperfect 
implementation (one teacher at one secondary school received training but taught a control class; at 
another secondary school, two teachers taught intervention classes without having had the initial 
SRSD training) and despite being used for a relatively short period of time (less than a single 
academic year in total). 
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Future research and publications 


Given the large effect size and positive feedback from teachers, we would recommend a much larger- 
scale effectiveness RCT of the intervention in undertaken to confirm the findings of this trial. The 
rationale behind this is as follows. First, the trial was geographically confined and focused on ‘transfer’ 
or ‘transition’ pupils. Second, Calderdale teachers adapted the intervention to suit their context and a 
formal evaluation of training teachers in an ‘Anglicised’ version would be appropriate. We consider that 
a trial of an Anglicised version of SRSD aimed at all primary school pupils with KS2 English 
assessments as the main outcome would be useful. Additionally, a trial of teaching the technique to 
secondary school teachers and using GCSE English scores as the main outcome would also be a 
valuable study. These two trials could be run in parallel. 
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Appendices 


Appendix A: Trial Diagram 


Primary Schools Recruited n = 24 


Secondary Schools Recruited n = 3 


Children recruited n = 288 
(based on 12 children per school) 
Inclusion criteria: 


e Yr6 

e predicted to achieve Level 3 or an insecure level 4 
in English by the end of Key Stage 2 (based on 
teacher assessments conducted at the end of 


Autumn term 2012) Excluded 


e Not meeting inclusion criteria 
e Other reasons 


Baseline data collection Feb 2013 


Key stage 2 English Teacher Assessments from Dec 
2012 


Primary School - 
Cluster 
ate laceceyanieesetice) al 


Control Group Intervention Group 


Schools N = 12 Schools N = 12 


Children N =144 Children N =144 


No intervention Package based on SRSD. Memorable 
experiences during Summer term 2013. 
Continued intervention in Year 7 in Secondary 


Schools). 


(Primary Schools will 
receive intervention next 
academic year) 


Follow up data collection Dec 2013 


Progress in English 11 (long form) 
(Conducted in Secondary School) 


Long term follow up 


Routine test results recorded in 
National Pupil Database 
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Appendix B: Trial Timelines 


No) ef=Vate | fer-t<) 


Calderdale Timeline 


Apply for University Ethics 
Intervention Development 
Recruitment of Schools 
Baseline Data Collection 
Randomisation 

Delivery of Intervention 
Post Intervention Testing 
Analysis & Report Writing 


Dec-12 Jan-13 Feb-13 Mar-13  Apr-13 May-13 Jun-13 


Jul-13__Aug-13 Sep-13 Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14_ Apr-14 
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Appendix C: 


Expression of Interest 
Document for Schools 


THE UNIVERSITY Of York 


AB 
Wy Durham 


University 


Evaluation of Calderdale Excellence 
Partnershi Improvin Writin Qualit 
intervention 


The Education Endowment Foundation has asked 
researchers at the University of York and Durham 
University to evaluate the Improving Writing Quality 
intervention being delivered and supported by the 
Calderdale Excellence Partnership in 2013 and 
2014. In addition to providing memorable 
experiences for the pupils the intervention will 
include professional development for teachers in 
key elements of the writing intervention Self- 
Regulated Strategy Development (SRSD), 
including discussion, modelling and planning. The 
evaluation aims to find out if the intervention helps 
to improve pupils’ writing skills during the transition 
from Year 6 to Year 7, especially the writing skills 
of pupils who are currently working at KS2 Level 3 
or a fragile KS2 Level 4. 


All primary schools who decide to take part in the 
intervention and its evaluation will be asked to 
provide information to Durham University about all 
pupils currently in Year 6 at their school. The 
information will include each Year 6 pupil’s UPN 
and their KS2 teacher assessment (from the end of 
the autumn term 2012) plus some basic 
demographic information about the school. 


Primary schools will then be randomly allocated in 
March 2013 by an independent researcher at the 
University of York to either implement the 
intervention in summer term 2013 with their Year 6 
pupils or wait to implement the intervention in 
summer term 2014 with Year 6 pupils. Notice will 
be taken of the demographic information to ensure 
that the two groups are balanced. Schools 
delivering the programme in 2013 will receive 
training in March 2013. The intervention will be 
delivered to all Year 6 pupils in the experimental 


group. 


Pupils taking part in the intervention will continue to 
receive the intervention when they move into Year 
7. All secondary schools who decide to take part 
will be asked to place all pupils from primary 
schools who have been running the programme 
into one half of Year 7 (who will continue to take 
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Appendices 


part in the activities in the autumn term 2013), and 
all pupils from primary schools who have not been 
running the programme in 2013 into the other half 
of Year 7 (who will receive the usual teaching in 
autumn term 2013). The intervention will be 
delivered to those classes/sets in which there are 
pupils from the intervention schools. 


At the end of autumn term 2013 all Year 7 pupils 
(with some _ exceptions where needed) in 
participating secondary schools will complete the 
GL Assessment Progress in English (PiE) 11 (Long 
Form). Secondary schools will send the pupils’ 
results to Durham University. 


The PiE results of all pupils taking part in the 
intervention will then be compared with the results 
of all children receiving the usual teaching, in order 
to estimate the effect the intervention has had on 
pupils’ writing skills. 


What commitment would this project require 
from schools? 


e Enthusiasm for the project and for your own 
professional learning 

e Provision of baseline data about pupils in Year 6 
(primary schools) 

e Willingness to allow random allocation to the 
‘Memorable Experiences’ intervention in 2013 or 
2014 (primary schools) 

e Willingness to place all pupils from primary 
schools implementing the intervention in 2013 in 
one half of Yr 7 and all pupils from primary schools 
waiting to implement the intervention in 2014 in the 
other half of Yr 7 (Secondary schools) 

e Attendance at professional development days 

e Willingness to implement the intervention 

e Willingness to administer a writing test to all Year 
7 pupils in December 2013 (secondary schools) 
and to provide the results to Durham University 
(the test will be marked independently by external 
markers) 
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When will this project take place? 


We hope to randomise participating schools in 
early March 2013. The first CPD days will be in 
March 2013. For primary schools allocated to 
implement the intervention in 2013, the activities 
will begin after the Easter holidays. 


Is there funding to support my involvement? 


Yes — funding for memorable experiences, training 
and development. 


Please come to the information meeting to find 
out more 


On Tuesday 26" February 2013 the Calderdale 
Excellence Partnership and the Evaluation team 
will jointly hold an information meeting for schools 
to find out more about the intervention and its 
evaluation. We very much hope to see you at this 
event — 1.15pm for a 1.30pm start and a 3.00pm 
finish at the Cedar Court Hotel, Lindley Moor Road, 
Ainley Top, Huddersfield, West Yorkshire HD3 
3RH. To book a place, please contact the 
Partnership office. Email: hxec.co.uk Tel: 01422 
255006 


Appendices 
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Appendix D: Primary School Agreement to participate form 


ae ones sine 
Calderdale Excellence Straten ucation 
THE UNIVERSITY Of York WPurham aan 


Foundation 


Evaluation of Calderdale Excellence Partnership /mproving Writing Quality intervention 


Primary School Agreement to Participate 


| confirm that | have read and understood the information sheet for the above evaluation and 
have had the opportunity to ask questions; 


| understand that all children’s results will be kept confidential and that no material which 
could identify individual children or the school will be used in any reports of this evaluation; 


| agree to send an information letter out to all parents/carers of children in Year 6 and collect 
in any returned opt out forms; 


| | | agree to provide baseline data about pupils in Year 6 to the evaluation team, Calderdale 
Excellence Partnership and EEF(excluding any pupils for whom opt out forms have been returned) 


| understand that named baseline data will be matched with the National Pupil Database and 
| | shared between the evaluation team, Calderdale Excellence Partnership and EEF. 


| | | agree to random allocation to implement the ‘Improving Writing Quality’ intervention in 2013 
or 2014 


| | agree to staff attending professional development days 


[| 


| consent to the school taking part in the above study. 


Name of headteacher 
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School Tel no 


Headteacher Email address 


Name of School Contact (if not 
N@ACLEACHESN) 1 on aciacad sosaraicaecatdadygh ens dab ean dd dames dab eesstasinmiduaeeesieeaemnanaacss 


School Contact email 
MNCS Sagas aprepemepiceecie dpacteces serene pacts tsnvnrt yuh encased Wa eoeees Aereng eee ss Aen ts apn eet tes Seneca 


Signature of 
NEAAtOACHES asi csisesdsesedvcdekancesne eaguwadasavawcugecas asaveeceddaee dayeeeaeesenses Dates iiciccvccennaieaasenccedas 


Thank you for agreeing to take part in this research. Please return this consent form at the information 
meeting or afterwards by post to: 


Calderdale Excellence Partnership office Room 121, E Mill, Dean Clough, Halifax, HX3 5AX 
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Appendix E: Secondary School Agreement to participate 
form 


rT | — 
Iderdale Excellence (ttatannty ducation 
THE UNIVERSITY OfYork WDurham “wens ‘eal rE ene 


Foundation 


Evaluation of Calderdale Excellence Partnership /mproving Writing Quality intervention 


Secondary School Agreement to Participate 


| confirm that | have read and understood the information sheet for the above evaluation and 
have had the opportunity to ask questions; 


| understand that all children’s results will be kept confidential and that no material which 
could identify individual children or the school will be used in any reports of this evaluation; 


| agree to place all pupils from primary schools implementing the intervention in 2013 in one 
half of Yr 7 and all pupils from primary schools waiting to implement the intervention in 2014 in 
the other half of Yr 7; 


| agree to implement the ‘Improving Writing Quality’ intervention with children in Year 7 from 
primary schools implementing the intervention in 2013; 


| agree to staff attending professional development days 


| agree to send an information letter out to all parents/carers of children in Year 7 and collect 
in any returned opt out forms; 


| agree to administer a writing test to all Year 7 pupils in December 2013; 


oO OU U 


| understand that pupils’ test responses, date of birth and gender will be collected by GL 

Assessment and accessed by the evaluation team. Named data will be matched with the 

National Pupil Database and shared between the evaluation team, Calderdale Excellence 
Partnership and EEF (Excluding data on any pupils for whom opt out forms have been returned). 


= 


| consent to the school taking part in the above study. 
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Name of headteacher 


Name of School Contact (if not 
NeAadteacher) . icc. tewctavencihecs tectveecins dies ke dex ivanlaond ied avs beanie iad ae eddaandgiy aviaauaetanaand acandlaendeevenes 


School Contact email 
72 (0 [0 [ots ae ea ea Se Ie ORS SE Se ee SO 


Signature of 
NEAGLEACHE|)R i isis snntenecnncnvcdancadnndnaastaiumiwednennsaiaubmeveventany ceeded evenexendanseNanmenwadneantehean Date..... 


Thank you for agreeing to take part in this research. Please return this consent form at the information 
meeting or afterwards by post to: 


Calderdale Excellence Partnership Office Room 121, E Mill, Dean Clough, Halifax, HX3 5AX 
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Appendix F: Parent and Pupil Information Letter Year 6 


ae 4 , é Educati 
Calderdale Excellence Straten ucation 
THE UNIVERSITY Of York WPurham a ee 


Foundation 


[INSERT DATE] 
[INSERT SCHOOL NAME] 
Dear Parent / Carer 


Your child’s school is taking part in the Improving Writing Quality programme evaluation. Durham 
University and the University of York have been asked by the Education Endowment Foundation (an 
organisation funding research into education) to independently evaluate the /mproving Writing Quality 
programme. 


The Improving Writing Quality programme is being led by the Calderdale Excellence Partnership (an 
organisation which helps schools in the Calderdale area). It is designed to improve children’s writing 
skills during the transition from Year 6 to Year 7. Good writing skills are important for all children. 


To find out how well the Improving Writing Quality programme works some schools will use the 
Improving Writing Quality programme this year and some schools will not. This is decided randomly 
by a computer. (However all schools will continue to teach children writing skills.) Researchers will 
then compare results from schools that have used the programme with schools that have not. In order 
to do this we would like to collect information about your child from your child’s primary school and 
later from your child’s secondary school. 


Your child’s school will provide information including your child’s name, date of birth, gender, unique 
pupil number, details on your child’s current National Curriculum writing level and free school meal 
status. 

Your child’s information will be treated with the strictest confidence. Named data will be matched with 
the National Pupil Database and shared between the evaluation team, Calderdale Excellence 
Partnership and EEF. We will not use your child’s name or the name of the school in any report 
arising from the research. Your child’s information will be kept confidential at all times. 


If you are happy for your child’s information to be used you do not need to do anything. Thank 
you for your help with this project. 


If you would rather your child’s school did not share your child’s information for this project please 
complete the enclosed opt out form and return it to your child’s school by [INSERT DATE]. 


If you would like further information about the Improving Writing Quality evaluation please contact 
Hannah Ainsworth the Evaluation Coordinator: hannah.ainsworth@york.ac.uk; 01904 328158 


Yours faithfully 


Professor David Torgerson (University of York) 
Professor Carole Torgerson (Durham University) 


Calderdale Excellence Partnership 
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Education Endowment Foundation 


ae | Dd Educati 
lerdale Exceller ; ucation 
THE UNIVERSITY Of York W Durham ul ii Fe eae 


Foundation 


Improving Writing Quality Evaluation: Opt Out Form 


If you DO NOT want your child’s data to be shared for use in the Improving Writing Quality evaluation, 
please return this form to your child’s school asap. 


| DO NOT want my child’s data to be shared for use in the Improving Writing Quality evaluation 


Ld 


Parent/Carer DIGMALU ES i iexeu tine sedawadaaeenine mitaclenanceu te mavielnn caentdhamideneamnaneeede ead ated 


Child’s 
A 2 (6 © NRE bo ee Ret en mn EP oe ee Ee pe 
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Appendix G: Parent and Pupil Information Letter Year 7 


AB = 7 - r Educati 
Calderdale Excellence Straten ucation 
THE UNIVERSITY O/York WPurham FO ae 


Foundation 
[INSERT DATE] 
[INSERT SCHOOL NAME] 
Dear Parent / Carer 
Your child’s school is taking part in the Improving Writing Quality programme evaluation. Durham 
University and the University of York have been asked by the Education Endowment Foundation (an 
organisation funding research into education) to independently evaluate the Improving Writing Quality 
programme. 
The Improving Writing Quality programme is being led by the Calderdale Excellence Partnership (an 
organisation which helps schools in the Calderdale area). It is designed to improve children’s writing 
skills during the transition from Year 6 to Year 7. Good writing skills are important for all children. 
To find out how well the /mproving Writing Quality programme has worked we would like to use 
English test results. Your child will do a test at school designed by GL assessment. When the test is 
completed the test will be sent to GL assessment who will mark the test and send the test results to 
the evaluation team who will analyse the results of the test as part of the research. Your child’s 
gender and date of birth will also be collected. 
Your child’s information will be treated with the strictest confidence. Named data will be matched with 
the National Pupil Database and shared between the evaluation team, Calderdale Excellence 
Partnership and EEF. We will not use your child’s name or the name of the school in any report 
arising from the research. Your child’s information will be kept confidential at all times. 


If you are happy for your child’s information to be used you do not need to do anything. Thank 
you for your help with this project. 


If you would rather your child’s school did not share your child’s information for this project please 
complete the enclosed opt out form and return it to your child’s school by [INSERT DATE]. 


If you would like further information about the Improving Writing Quality evaluation please contact 
Hannah Ainsworth the Evaluation Coordinator: hannah.ainsworth@york.ac.uk; 01904 328158 


Yours faithfully 


Professor David Torgerson (University of York) 
Professor Carole Torgerson (Durham University) 
Calderdale Excellence Partnership 


Education Endowment Foundation 
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Ae Calderdale Bycellenae | fi Ed ti 
Calderdale Excellence ucation 
THE UNIVERSITY Of York WW Durham Esl Ee 


Foundation 


Improving Writing Quality Evaluation: Opt Out Form 


If you do not want your child’s data to be shared for use in the Improving Writing Quality evaluation, 
please return this form to your child’s school asap. 


lke want my child’s data to be shared for use in the Improving Writing Quality evaluation 


Parent/Carer DIGMAWNC iesiadiriedrerdeosnelecheebiaskde chinaghdom dna apeeberdhineaehdeeedas 


Child’s 
LS Yo] (615 eens Re Pt RSPCA eave SPOS te eon Pe Oe Ea eS a TE ena re ee nee eS 
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Appendix H: Variable Collection Table 
DP = Delivery Partner (Calderdale) 
GL = GL Assessment 


EVY = Evaluators (York) 


EEF = Education Endowment Fund 
EV = Evaluators (York and Durham) 


EVD = Evaluators (Durham) 


otherwise stated) 


Pupil 
Information: 


Post 
Intervention: 


Variable 
School 
Information: School Name 
(For both primary and Secondary School DCSF code 
secondary schools Secondary School Address + Postcode 
unless 


Secondary School Telephone Number 

Secondary School Assessment Lead Name 

Secondary School Assessment Lead Email and Role 
Number of tests required per secondary school 

Number of Pupils on Roll 

% pupils FSM (Ever 6) 

% pupils from minority ethnic groups 

% pupils supported at school action plus without SEN statement 
Special measures (binary variable) 

Pupil Name 

UPN 

DoB 

Gender (M/F) 

Predicted KS2 English Level (end of Autumn Term 2012) 
Actual KS2 English Level 

FSM current 

FSM not ever 6 

PP 

EAL 

Secondary School Class/Registration Group 


Pupil post test results 
Fidelity Measure? 


Required By Comments 

All 

GL Currently missing (DP to collect) 

GL Currently missing (DP to collect) 

GL Currently missing (DP to collect) 

GL Currently missing (DP to collect) 

GL Currently missing (DP to collect) 

GL Currently missing (DP to collect) 

EV Currently missing for secondary schools 
EV Currently missing for secondary schools 
EV Currently missing for secondary schools 
EV Currently missing for secondary schools 
EV Currently missing for secondary schools 
EER EVDAGE 

EER EVDAGE 

EER EVAGIE Currently missing (Retrieve from NPD) 
EER EVAGE Currently missing (Retrieve from NPD) 
EER EV, 

EERE Retrieving from NPD 

EERVEV, 

EERE, 

EER EV, Currently missing (Retrieve from NPD) 
EERE, Currently missing (Retrieve from NPD) 
EV, GL Currently missing (DP to collect) 

Bele, ev Provided by GL to EVD 

EV In development 
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Appendix I: Classroom Observation Information Sheet and 
Consent Form 


ae | oe 
Calderdale Excellence (@tratassitty f ducation 
THE UNIVERSITY OfYork “W)urham bates ae 


Foundation 


Improving Writing Quality Evaluation: Classroom Observation Information Sheet 


The Education Endowment Foundation has asked researchers at the University of York and Durham 
University to evaluate the Improving Writing Quality intervention being delivered and supported by the 
Calderdale Excellence Partnership in 2013 and 2014. 


As your school is taking part in the /mproving Writing Quality intervention we would like to observe you 
delivering a session of the Improving Writing Quality intervention to a class of pupils. 


We will use the observation to help us better understand the /mproving Writing Quality intervention 
and evaluate its use in schools. Neither yourself, your school, or any pupils will be identified in any 
reports resulting from the evaluation. 


It would be helpful to us if we could observe you delivering the Improving Writing Quality intervention 
to a class of pupils but you do not have to agree to be observed. If you decide you do want to take part 
then change your mind, you are free to withdraw at any time, without giving a reason. This will not 
affect your school taking part in the project generally. 


If you decide you would like to take part, please sign and return the consent form to the researcher. 
If you would like any more information please contact Hannah Ainsworth. 

Tel: 01904 328158 

Email: hannah.ainsworth@york.ac.uk 


Thank you for considering taking part 
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ae | he siete 
Iderdale Excelle ducation 
THE UNIVERSITY Of York WDurham “wes = Bonen 


Foundation 


Improving Writing Quality Evaluation: Classroom Observation Consent Form 


Please initial 
each box 
| confirm that | have read and understood the Improving Writing Quality 
Evaluation: Classroom Observation Information Sheet. | have had the 
opportunity to consider the information and ask questions. 


| understand that my participation is voluntary and that | am free to withdraw at 
any time, without giving any reason. 


| understand that my details will be provided to researchers working on the 
evaluation and will be stored securely. | understand that neither I, nor my 
school, nor any pupils will be identified in any reports resulting from the 
evaluation. 


| agree to be observed delivering a session of the /mproving Writing Quality 
intervention to a class of pupils. 


PF} LU wu 


Teacher 
VIG MAU Gs 2 sa Anata dat ances eeein dertesnc ae enteeee sain ge ielenenae teed alate eten ares oa dedi stan ae oe eat tea on pana anenre tates 


Thank you for taking part 
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Appendix J: Classroom Observation Schedule 


Improving Writing Quality Evaluation 


Observation Schedule 


Introduction 

Introduce yourself 

Talk through Information Sheet for observation 

Provide opportunity for questions about the observation 


Confirm consent to take part. 

Observation 

Date: —........4. | Peer rere | Peres 

Scoring System: 

1 —No/None 

2 — Poor explanation/Partly/To some degree/A few 


3 — Satisfactory explanation /Most 


4 — Yes/Good explanation/All 
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Observation Score Notes 


Was there a discussion about 
the writing genre being covered 
in the session? 


Did the teacher remind children 
to use the mnemonic IPEELL? 


Did children use the mnemonic 
IPEELL? 


Did the teacher remind children 
to use graphic organiser or 
planning sheet? 


Did the children use graphic 
organiser or planning sheet? 


Was a form of self-scoring 
used? 


Did the teacher model positive 
self-talk? 


Did the teacher encourage 
children to stay motivated by 
talking themselves through it? 


Did peer-scoring occur? 


Other Notes 


Thank teacher for allowing session to be observed 
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Appendix L: Teacher Interview Information Sheet and 
Consent Form 


J rp Educati 
lerdale Excellence Straten ucation 
THE UNIVERSITY of York WDurham ee ae 


Foundation 


Improving Writing Quality Evaluation: Teacher Interview Information Sheet 


The Education Endowment Foundation has asked researchers at the University of York and Durham 
University to evaluate the Improving Writing Quality intervention being delivered and supported by the 
Calderdale Excellence Partnership in 2013 and 2014. 


As your school is taking part in the /mproving Writing Quality intervention we would like to hear about 
your experiences of delivering and using the intervention with your pupils. 


We would like to conduct an interview with you, which will take approximately 30 minutes to complete. 
We would ask for your permission to record the interview and for the recording of the conversation to 
be typed into a transcript by an approved transcriptionist. 


We will use the information you provide to help us evaluate the Improving Writing Quality intervention. 
You will not be identified in any reports resulting from the evaluation. 


As your school is taking part in the /mproving Writing Quality intervention it would be helpful to us if we 
could get feedback from you but you do not have to take part in an interview. If you decide you do 
want to take part then change your mind, you are free to withdraw at any time, without giving a reason. 
This will not affect your school taking part in the project generally. 


If you decide you would like to take part, please sign and return the consent form to the researcher. 
If you would like any more information please contact Hannah Ainsworth. 

Tel: 01904 328158 

Email: hannah.ainsworth@york.ac.uk 


Thank you for considering taking part 
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{4 d Educati 
Iderdale Excelle ucation 
THE UNIVERSITY Of York WDurham “wes = Bonen 


Foundation 


Improving Writing Quality Evaluation: Teacher Interview Consent Form 


Please initial 
each box 

| confirm that | have read and understood the Improving Writing Quality 

Evaluation: Teacher Interview Information Sheet. | have had the opportunity to 

consider the information and ask questions. 


| understand that my participation is voluntary and that | am free to withdraw at 
any time, without giving any reason. 


evaluation and will be stored securely. | understand that my personal details will 
be removed from any reports resulting from the evaluation. 


| agree that | will take part in the Improving Writing Quality Evaluation interview. 
| give permission for the interview to be audio recorded and transcribed. 


| understand that direct quotations may be used in publications but no 
information will be released or printed that would identify me 


| understand that my details will be provided to researchers working on the | | 


Teacher 
IGM AUC as as ca cinicatie wodrwciinndnoniad ie daira Rie Rote oicaan cam ered een omen eearide eb ietionaxeiad ancien 


Thank you for taking part 
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Appendix M: Teacher Interview Topic Guide 
Improving Writing Quality Evaluation 


Topic guide for interviews with teachers 


Introduction 

Introduce yourself 

Talk through Information Sheet for the interview 
Provide opportunity for questions about the interview 


Confirm consent to take part. 


Experience of the Improving Writing Quality intervention (other names: SRSD Writing Project, 
Passing the Baton) 


Training 
What training did you receive? 


How did you find the training? 


Did you use any other resources? 
e CASL website 


e Any of the books recommended 


Support 
Thoughts on support offered by CEP during intervention phase. 


Any other support required or would be helpful 


Classroom delivery 
Experiences of using the intervention in the classroom 


What worked well 


50 


Appendices 


What worked less well 


Memorable Experiences 
Please can you tell me about any memorable experiences the pupils took part in? 


Did these help pupils> How and Why? 


Impact 

Thoughts on the impact of the intervention on pupils writing skills 
Quality of Writing 

Knowledge of Writing 

Approach to writing 


Self-efficacy 


End interview 


Is there anything else you want to tell us about the Improving Writing Quality intervention or taking part 
in the independent evaluation? 


Do you have any questions? 


Thank participant 
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Appendix N: Essential Elements of Intervention (provided 
by CEP) 


Self-Regulated Strategy Development 


Primary schools - Essential elements: 


NB It is important to deliver SRSD in the order outlined in the manual but it is acceptable to combine 
lessons rather than stick rigidly to the prescribed model. 


e Discussion about genre 
> Non -—narrative genres only in primary schools: 


Persuasive writing 
Recount 

Balanced argument 
Explanation 
Instructional writing 


00000 


e Pre assessment — all children should do a piece of writing in each genre before 
starting teaching using SRSD. 


> Tobe carried out by class teachers before starting the SRSD programme. 
> There is no need for these assessments to be sent to the CEP office although it 
would be helpful to keep copies so that they can be used in future training. 


e Mnemonics 


> Mnemonics should be used to help pupils structure their writing. 
> Atprimary level the mnemonic to use is IPEELL. 


| = Introductory paragraph; 

P= Points; 

E = Examples/elaboration; 

E = End; 

L = Links (connectives, openers); 


L = Language (wow words, genre specific vocabulary, punctuation) 
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Graphic organisers 


> Children should use a graphic organiser or planning sheet to help structure their 


writing: 

Introductory 

paragraph 
Point + Examples Point + Examples Point + Examples 

Ending 
Links Language 
(connectives, (genre specific 
openers) vocab, wow words) 


e Self-scoring and graphing 


> Devise your own self scoring system for pupils to use based on any of the 
examples in the manual. (Please see the example) 

> Children should practise scoring their own work against an agreed scoring 
schedule. 

> They should be encouraged to mark using “two stars and a wish” (2 good points 
and something to improve on the next piece of work). 

> Always graph results. 


e Self-talk: Teachers should model positive self-talk and encourage children to stay 
motivated throughout the writing task by talking themselves through it. 


> Use terminology you are comfortable with: “Motivational messages’, “Positive talk” 
> Use self-talk at all stages 


e Peer scoring 


> Children should work in pairs to check each other’s work using the marking 
schedule. 
> Pupils should explain their judgements. 


e Final Assessment 


> Acold task to be carried out one/two weeks after the end of the unit. 
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Appendix O: Ceiling/Floor Effect Examination 


The test and outcomes were assessed for ceiling or floor effects using histograms and summary 
statistics. 


e Ceiling effects occur if a test is too easy - i.e. lots of pupils achieve a perfect score. 
e Floor effects occur if a test too hard — i.e. lots of pupils score zero. 


There does not appear to be a strong floor or ceiling effect in overall raw score, extended writing, 
reading or spelling or grammar scores. 


Overall raw score 


Overall raw score can range between 0 and 84 marks. The histogram below shows the distribution of 
post test scores achieved and summary statistics are presented in the Table below. The mean score 
was slightly lower than the median score suggesting a slight skew to the data which can be seen in 
the histogram. The minimum score observed was 20 out of 84, no pupil achieved full marks. The 
mode score was 62 marks. As seen in the histogram, there was not a large proportion of pupils 
achieving particularly high or low score implying no ceiling or floor effect. 


Overall Min 1*Q Med 3°Q Max Mean SD 
Raw 
Score 20 43 52 60 79 51.0 11.71 


Distribution of Overall Raw Post-Test Score 


05 
! 


20 40 60 
Calculated raw score after assumptions applied 
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Histograms of overall raw score by predicted level (below) were produced and it appears that the test 
distinguished between different ability levels. 


Level 3 Level 4 


05 


2 vt 


Level 5 or above 


Calculated raw score after assumptions applied 


____ Density 


normal rawscore 


Graphs by reducedbase 


Primary outcome (extended writing) 


The primary outcome of extended writing score can range between 0 and 32 marks. The histogram 
below shows the distribution of post-test extended writing scores achieved and summary statistics are 
presented in the Table below. The median and mean scores were similar at around 23 marks 
suggesting non-skewed data. The minimum score observed was 8 out of 32; the maximum score was 
32 marks. The mode score was 24 marks. As seen in the histogram, there was not a large proportion 
of pupils achieving particularly low score and only 12 pupils achieved full marks implying no ceiling or 
floor effect. 


Extended Writing | Min 1*Q@ Med 3°Q = Max Mean SD 
Score 8 18 23 25 32 22.0 5.07 
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Distribution of Extended Writing Score 


10 20 30 
Sum of raw scores from the two extended writing tasks (Q5 & Q6) 


Secondary outcome — Reading 


The secondary outcome of reading score can range between 0 and 32 marks. The histogram below 
shows the distribution of post-test reading scores achieved and summary statistics are presented in 
the Table below. The median and mean scores were similar at around 16 marks implying no skew. 
The minimum score observed was 3 out of 32; the maximum score was 29 marks. The mode score 
was 21 marks. As seen in the histogram, there was not a large proportion of pupils achieving 
particularly high or low score implying no ceiling or floor effect. 


Reading Min 1*Q Med 3%Q Max Mean SD 
Score 3 12 16 21 29 16.5 5.07 


Distribution of Reading Score 


30 


10 20 
Sum of raw scores from Q3, Q3x, Q4 & Q4x 


Secondary outcome - Spelling and Grammar 


Appendices 


The secondary outcome of spelling and grammar score can range between 0 and 20 marks. The 
histogram below shows the distribution of post-test spelling and grammar scores achieved and 
summary statistics are presented in the Table below. The median and mean scores were similar at 
around 11 marks suggesting there is not a great concern over scores following a highly skewed 
distribution. The minimum score observed was 0 out of 20; the maximum score was 20 marks. The 
distribution was bi-modal with modes at 12 and 14 marks. As seen in the histogram, there was not a 
large proportion of pupils achieving particularly high or low score implying no ceiling or floor effect. 


Spelling and 
Grammar Score 


Min 
0 


1*Q@ Med 37°Q Max 


6 


11 14 20 


Mean 
10.2 


sD 
5.06 


Distribution of Spelling and Grammar Score 


5 10 1 
Sum of raw scores from Q1 & Q2 
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Appendix P: Coefficients and standard errors for effect 


Sizes 
Coefficient SE af Residual SD SE of Residual 
(relating to Coaticiant (Random SD 
intervention) Effect) 
L3c — L4b 
Primary analysis 2.5299438 0.83200796 3.4226462 0.1648204 
Secondary (reading) -0.30961708 0.8751381 3.4454817 0.1931986 
Secondary (spelling) -0.44108015 0.65325476 3.3753853 0.1701447 
L4a+ 
Primary outcome 0.00017931 0.70502193 3.0987022 0 .1859449 
Secondary (reading) -0.38720413 0.71993268 3.1076949 0 .1946564 
Secondary (spelling) 0.10404381 0.78438441 2.9343483 0.1733643 
L3c+ 
Primary outcome 0.50696715 0.68195886 3.392538 0.1213736 
Secondary (reading) -0.3435145 0.55783096 3.3802348 0.13374 
Secondary (spelling) -0.13289695 0.65894462 3.1632813 0.1134944 
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Appendix Q: Coefficients relating to the primary analysis 


Variable Coefficient (95% Cl) p-value 
Allocation 

Control Reference group - 
Intervention 2.53 (0.90 to 4.16) 0.002 
Eligible for FSM 

No Reference group - 

Yes -0.24 (-1.21 to 0.73) 0.628 
English as additional 

language 

No Reference group - 

Yes 2.05 (0.71 to 3.40) 0.003 
Gender 

Female Reference group - 

Male -2.47 (-3.40 to -1.54) <0.001 
Month of Birth 

Jan Reference group - 

Feb 1.44 (-0.59 to 3.47) 0.164 
Mar 1.91 (-0.20 to 4.02) 0.076 
Apr 1.56 (-0.59 to 3.70) 0.155 
May 2.65 (0.58 to 4.71) 0.012 
Jun 1.28 (-0.67 to 3.23) 0.198 
Jul 0.58 (-1.62 to 2.78) 0.608 
Aug 1.53 (-0.65 to 3.71) 0.168 
Sep 2.21 (-0.06 to 4.48) 0.056 
Oct 0.19 (-1.88 to 2.26) 0.858 
Nov 0.88 (-1.37 to 3.14) 0.443 
Dec 2.34 (0.11 to 4.57) 0.04 
Predicted KS2 level 

Level 3c Reference group - 
Level 3b 0.11 (-2.21 to 2.43) 0.928 
Level 3a 3.15 (0.62 to 5.67) 0.014 
Level 4c 4.15 (1.91 to 6.39) <0.001 
Level 4b 5.31 (3.04 to 7.59) <0.001 
Constant 14.72 (11.82 to 17.62) <0.001 
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Appendix R. Security rating summary 


1.Design: ae 3. Attrition: 4. Balance: a= 
What is the 53 What is the Adjustment ae 
aalialianlelan Adjustment to 


uality of the AV) re) meg to rating t 
q : Y detectable ae Pe account for issues 
design of the out from the account for : 

effect at the with 


evaluation? evaluation? balance? : : 
start? Taie-ise)e-1e-14 (0) 0g 


Evaluation design Implementation Analysis and interpretation 


Fair and clear experimental 
design (RCT, RDD) 


Well-matched comparison 
(quasi-experiment) 


Matched comparison 
(quasi-experiment) 
Comparison group with 
poor or no matching 


The final security rating for this trial is 2. This means the findings are of moderate security. The trial was 


designed as a small-scale, independently-run, efficacy trial that could have achieved a maximum of 2. This was 
achieved. It was run over a relatively short time period in 23 primary schools and three secondary schools, which 
reduces the generalisability of the findings. 


Attrition was low at below 10%. There is some threat to the validity of the results as tests were delivered by 
schools, rather than blind by evaluators. However, there is no evidence to suggest they were not administered 
under exam conditions as required. There is also some threat from contamination caused by teachers sharing 
practice, because both treatment and control children attended the same secondary schools. 


However, the main concern is that the precision of the estimate is weak due to the low power. To increase the 
security and generalisability of the findings, a further trial at a larger scale could be undertaken. 


Additional detail about the EEF’s security rating system can be found at: 


www.educationendowmentfoundation.org.uk/evaluation. 
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